Skip to main content

Survey Crawls

Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.



rss RSS

PART OF
Internet Archive Web Crawls
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
collection
eye 1.4B
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection
eye 973M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Survey Crawl Number 6: Sep 11th, 2017 - running now
collection
13,723
ITEMS
751.8M
VIEWS
collection
eye 751.8M
The seeds for this crawl came from: 251 million Domains that had at least one link from a different domain in the Wayback Machine, across all time ~ 300 million Domains that we had in the Wayback, across all time 55,945,067 Domains from https://archive.org/details/wide00016 This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection
eye 736M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection
eye 545.3M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Survey Crawl Number 8
collection
8,699
ITEMS
527.8M
VIEWS
collection
eye 527.8M
Survey Crawl Number 7
collection
6,605
ITEMS
515.8M
VIEWS
collection
eye 515.8M
This "Survey" crawl was started on Feb. 24, 2018. This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) Survey 7 is based on a seed list of 339,249,218 URLs which is all the URLs in the Wayback Machine that we saw a 200 response code from in 2017 based on a query we ran on Feb. 1st, 2018.   The WARC files associated with this crawl are not currently available to the general public.
collection
eye 468.6M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
.com survey started January 2011
collection
2,535
ITEMS
348.7M
VIEWS
collection
eye 348.7M
Survey crawl of .com domains started January 2011.
Topic: webcrawl
collection
eye 230.2M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Survey Crawl Number 9
collection
502
ITEMS
67.2M
VIEWS
collection
eye 67.2M
COM Survey Crawl 2009-2010
collection
729
ITEMS
55.5M
VIEWS
collection
eye 55.5M
COM survey crawl data collected by Internet Archive in 2009-2010. This data is currently not publicly accessible.
survey_net00000
collection
300
ITEMS
41.2M
VIEWS
collection
eye 41.2M
Survey crawl of .net domains started December 2010.
Topic: webcrawl
ORG Survey Crawls
collection
191
ITEMS
24.4M
VIEWS
collection
eye 24.4M
Survey of .org domains. This data is currently not publicly accessible.
survey_net00001
collection
170
ITEMS
15.3M
VIEWS
collection
eye 15.3M
Survey crawl of .net domains started October 2011.
Topics: webwidecrawl, net
Survey Crawl Number 8
web
eye 4.7M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl818.us.archive.org:survey from Fri Dec 14 21:00:08 PST 2018 to Fri Dec 14 22:34:06 PST 2018.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl344.us.archive.org:survey from Thu Oct 12 08:48:34 PDT 2017 to Thu Oct 12 01:56:31 PDT 2017.
Topic: crawldata
Survey Crawl Number 7
web
eye 2.4M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl-hq10.us.archive.org:survey from Sat Feb 24 03:26:41 PST 2018 to Fri Feb 23 19:55:17 PST 2018.
Topic: crawldata
Survey Crawl Number 7
web
eye 2.2M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl840.us.archive.org:survey from Tue May 15 20:39:06 PDT 2018 to Tue May 15 21:00:30 PDT 2018.
Topic: crawldata
Survey Crawl Number 7
web
eye 2.1M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl841.us.archive.org:survey from Mon May 14 17:51:22 PDT 2018 to Mon May 14 16:01:52 PDT 2018.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl835.us.archive.org:survey from Sun Jan 10 18:55:42 PST 2016 to Sun Jan 10 11:20:26 PST 2016.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl428.us.archive.org:survey from Thu Jan 8 02:39:52 PST 2015 to Thu Jan 8 02:17:55 PST 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl420.us.archive.org:survey from Mon Dec 23 10:49:15 PST 2013 to Mon Dec 23 05:58:22 PST 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl452.us.archive.org:survey from Sun Jan 25 16:21:37 PST 2015 to Sun Jan 25 09:51:07 PST 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl422.us.archive.org:survey from Sat Dec 20 22:29:58 PST 2014 to Sat Dec 20 16:52:21 PST 2014.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl429.us.archive.org:survey from Fri Jan 17 02:48:53 PST 2014 to Thu Jan 16 23:21:44 PST 2014.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl336.us.archive.org:survey from Fri Jan 17 01:34:59 PST 2014 to Thu Jan 16 20:57:28 PST 2014.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl419.us.archive.org:survey from Thu Dec 19 09:08:50 PST 2013 to Thu Dec 19 02:00:44 PST 2013.
Topic: crawldata
Survey Crawl Number 7
web
eye 1.2M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl-hq10.us.archive.org:survey from Sat Feb 24 10:06:32 PST 2018 to Sat Feb 24 02:13:22 PST 2018.
Topic: crawldata
Survey Crawl Number 8
web
eye 1.2M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl818.us.archive.org:survey from Sun Jan 27 10:04:49 PST 2019 to Sun Jan 27 06:59:10 PST 2019.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl419.us.archive.org:survey from Thu Jan 16 23:17:39 PST 2014 to Thu Jan 16 20:04:33 PST 2014.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl454.us.archive.org:survey from Thu Dec 12 08:45:40 PST 2013 to Thu Dec 12 01:43:13 PST 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl427.us.archive.org:survey from Wed Dec 31 23:34:59 PST 2014 to Wed Dec 31 21:31:58 PST 2014.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl804.us.archive.org:survey from Mon Feb 22 06:57:25 PST 2016 to Mon Feb 22 01:01:29 PST 2016.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl338.us.archive.org:survey from Wed Dec 31 23:29:37 PST 2014 to Wed Dec 31 21:16:42 PST 2014.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl425.us.archive.org:survey from Thu Jan 1 00:34:54 PST 2015 to Thu Jan 1 01:57:10 PST 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl426.us.archive.org:survey from Thu Jan 1 07:23:56 PST 2015 to Thu Jan 1 05:20:34 PST 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl423.us.archive.org:survey from Thu Jan 29 19:18:01 PST 2015 to Thu Jan 29 11:53:35 PST 2015.
Topic: crawldata
Survey Crawl Number 8
web
eye 1.1M
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl837.us.archive.org:survey from Thu Dec 6 15:19:06 PST 2018 to Thu Dec 6 13:31:02 PST 2018.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl804.us.archive.org:survey from Fri Dec 19 15:17:43 PST 2014 to Fri Dec 19 08:41:03 PST 2014.
Topic: crawldata
Survey Crawl Number 8
web
eye 946,210
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl835.us.archive.org:survey from Thu Jan 3 00:58:44 PST 2019 to Wed Jan 2 18:14:34 PST 2019.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl423.us.archive.org:survey from Sun Dec 21 20:51:55 PST 2014 to Sun Dec 21 16:00:10 PST 2014.
Topic: crawldata
Data crawled by Internet Archive on behalf of Internet Archive from Sat Jan 02 00:17:41 PDT 2010 to Sat Jan 02 00:40:59 PDT 2010
Topic: crawldata
Survey Crawl Number 6: Sep 11th, 2017 - running now
web
eye 922,148
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl344.us.archive.org:survey from Sun Oct 29 08:31:23 PDT 2017 to Sun Oct 29 02:17:27 PDT 2017.
Topic: crawldata
Survey Crawl Number 6: Sep 11th, 2017 - running now
web
eye 908,636
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl339.us.archive.org:survey from Sat Sep 30 04:27:26 PDT 2017 to Fri Sep 29 21:56:42 PDT 2017.
Topic: crawldata
Survey Crawl Number 7
web
eye 906,146
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl843.us.archive.org:survey from Sun Mar 4 03:11:53 PST 2018 to Sat Mar 3 21:10:41 PST 2018.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl453.us.archive.org:survey from Fri May 31 21:15:22 PDT 2013 to Sun Jun 2 06:31:08 PDT 2013.
Topic: crawldata
.com survey started January 2011
web
eye 865,447
favorite 0
comment 0
Internet Archive crawldata from COM zone survey, captured by crawl309.us.archive.org:com from Wed Feb 23 14:38:26 PST 2011 to Wed Feb 23 08:38:53 PST 2011.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl801.us.archive.org:survey from Mon Jan 26 09:41:07 PST 2015 to Mon Jan 26 15:41:07 PST 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl453.us.archive.org:survey from Wed May 22 00:29:45 PDT 2013 to Wed May 29 10:38:37 PDT 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl803.us.archive.org:survey from Sat Aug 1 00:46:17 PDT 2015 to Sat Aug 1 12:00:07 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl815.us.archive.org:survey from Sat Aug 1 00:44:26 PDT 2015 to Sat Aug 1 11:53:01 PDT 2015.
Topic: crawldata
Survey Crawl Number 7
web
eye 842,325
favorite 0
comment 0
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl847.us.archive.org:survey from Sun Mar 4 00:17:03 PST 2018 to Sat Mar 3 17:52:21 PST 2018.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl806.us.archive.org:survey from Sat Aug 1 00:46:36 PDT 2015 to Sat Aug 1 12:04:06 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl835.us.archive.org:survey from Sat Aug 1 00:43:36 PDT 2015 to Sat Aug 1 11:51:01 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl424.us.archive.org:survey from Sat Aug 1 00:49:20 PDT 2015 to Sat Aug 1 06:21:05 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl800.us.archive.org:survey from Sat Aug 1 00:47:24 PDT 2015 to Sat Aug 1 11:53:48 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl426.us.archive.org:survey from Sat Aug 1 00:48:47 PDT 2015 to Sat Aug 1 05:36:22 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl428.us.archive.org:survey from Sat Aug 1 00:47:49 PDT 2015 to Sat Aug 1 11:57:41 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl422.us.archive.org:survey from Sat Aug 1 00:39:59 PDT 2015 to Sat Aug 1 05:48:43 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl808.us.archive.org:survey from Sat Aug 1 00:45:26 PDT 2015 to Sat Aug 1 11:47:09 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl451.us.archive.org:survey from Sat Jun 1 17:43:44 PDT 2013 to Sun Jun 2 20:36:52 PDT 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl836.us.archive.org:survey from Sat Aug 1 00:43:43 PDT 2015 to Sat Aug 1 11:44:39 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl807.us.archive.org:survey from Sat Aug 1 00:45:46 PDT 2015 to Sat Aug 1 11:59:41 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl339.us.archive.org:survey from Sat Aug 1 00:38:24 PDT 2015 to Sat Aug 1 06:11:57 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl423.us.archive.org:survey from Sat Aug 1 00:49:35 PDT 2015 to Sat Aug 1 06:06:54 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl814.us.archive.org:survey from Sat Aug 1 00:44:34 PDT 2015 to Sat Aug 1 11:47:06 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl451.us.archive.org:survey from Sat May 25 23:20:37 PDT 2013 to Sun May 26 23:08:39 PDT 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl453.us.archive.org:survey from Tue Jun 18 10:36:57 PDT 2013 to Fri Jun 21 09:04:08 PDT 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl801.us.archive.org:survey from Sat Aug 1 00:47:17 PDT 2015 to Sat Aug 1 11:58:58 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl811.us.archive.org:survey from Sat Aug 1 00:44:50 PDT 2015 to Sat Aug 1 11:44:35 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl421.us.archive.org:survey from Sat Aug 1 00:39:52 PDT 2015 to Sat Aug 1 06:13:00 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl452.us.archive.org:survey from Sun May 26 01:33:43 PDT 2013 to Mon May 27 01:54:34 PDT 2013.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl802.us.archive.org:survey from Sat Aug 1 00:46:09 PDT 2015 to Sat Aug 1 11:55:45 PDT 2015.
Topic: crawldata
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl813.us.archive.org:survey from Sat Aug 1 00:44:39 PDT 2015 to Sat Aug 1 11:49:23 PDT 2015.
Topic: crawldata